Northeastern University in TREC 2009 Million Query Track

Ranking is a central problem in information retrieval. Modern search engines, especially those designed for the World Wide Web, commonly analyze and combine hundreds of features extracted from the submitted query and underlying documents in order to assess the relative relevance of a document to a given query and thus rank the underlying collection. The sheer size of this problem has led to the development of learningto-rank (LTR) algorithms that can automate the construction of such ranking functions: Given a training set of (feature vector, relevance) pairs, a machine learning procedure learns how to combine the query and document features in such a way so as to effectively assess the relevance of any document to any query and thus rank a collection in response to a user input. Much thought and research has been placed on the development of sophisticated learning-to-rank algorithms. However, relatively little research has been conducted on the construction of appropriate learningto-rank data sets nor on the effect of these data sets on the ability of a learning-to-rank algorithm to “learn” effectively. Given that the IR technology is ubiquitous in a vast variety of contexts and environments it is not unreasonable to assume that searchable material (corpora) and user information needs will radically vary from one retrieval environment to another. Theoretically, ranking functions should be trained over collections with similar characteristics as the collections they will be deployed in. However, the ability to construct different ranking functions for different retrieval environments is limited by the cost of constructing such customized training collections. Thus, the question that naturally arises is whether training on a collection of certain characteristics can still lead to an effective ranking function over collections of different characteristics. To answer this question we trained our ranking functions (by employing SVM) over two different collections, (a) the Million Query 2008 (MQ08) collection (GOV2 corpus and queries with at least one click on documents in the .gov domain), and (b) a Bing generated collection (described in Section 2.1) and employed the constructed ranking function over the Million Query 2009 (MQ09) collection (ClueWeb09 corpus and general web queries). Furthermore, even within a certain retrieval environment (represented by a given collection) different queries may have radically different characteristics and thus different features may better capture the notion of relevance. For instance, in the case of precision-oriented queries, such as homepage/namepage finding, the url of a document or its popularity may be more indicative of the document relevance than the document text itself, while for informational queries the document url maybe less indicative than its text. Most of the existing learning-to-rank approaches train a single ranking function to handle all queries. Hence, the question that arises is whether training a different ranking function for each one of these different query